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Abstract 


This paper describes the results of teacher education candidates’ use of a 
Web-based assessment system including electronic portfolios. A teacher 
education program adopted a novel Web-based Accountability Model for 
assessment and learning. The Web-based Education Accountability System 
(EAS) was based on the Web-based Accountability Model, and consisted 
of an electronic portfolio sub-system, an online performance assessment 
sub-system, and an Accountability Center sub-system. The objective of this 
study was to experimentally test teacher candidates’ views towards using the 
system. The experiment also examined the effect of the EAS on candidates’ 
cognitive learning. Thirty-one preservice teacher candidates enrolled in 
teacher certification programs participated in this study in Spring 2002. 
Questionnaires were presented to candidates as pre- and posttests. Content 
analysis was employed to qualitatively analyze the electronic por folio to 
examine the development of cognitive skills. The content analysis and 
questionnaire results showed that preservice teachers (1) thought the EAS 
positively facilitated learning to teach and helped them meet teacher stan- 
dards over time, and (2) became more engaged in reflective and cognitive 
activities by using the electronic por folio over time. 


D uring the last 1 0 years, performance-based assessment has been 
increasingly used to assess teachers’ competencies with the devel- 
opment of education theory from behaviorism to constructivism 
(Cannella & Reiff, 1994; Richardson, 1997). Performance assessment is 
the type of educational assessment in which judgments are made about 
student’s knowledge and skills based on observation of student behavior 
or inspection of student products (Lam, 1995). Most research in the 
context of PK-12 settings has 

• Explored the relationship between students’ achievements and 
teacher performance (Laczko-Kerr & Berliner, 1999) and 
• Developed general accountability models at the national, state, and 
district level (Lane, Parke, & Stone, 2002). 

Some researchers have employed accountability as an incentive mecha- 
nism by tying teachers’ compensation at least in part to performance 
(Mathews, 2004; Solmon & Podgursky, 2000). 

Portfolio assessment is widely adopted in American education. Nearly 
95% of schools and colleges use portfolios to make decisions about candidates 
(Herman, Aschbacher, & Winters, 1992; Salzman, Denner, & Harris, 2002; 
Wiggins, 1990). A portfolio is a representative collection of one’s work, 
providing documentation of work in progress, evidence of how work has 
evolved and how it has been refined (Amber & Czech, 2002). By collecting 
artifacts to annotate their progress, candidates articulate their learning and 
adjust their professional goals. By folfilling these intentions, candidates think 
and learn more because they are actively and willfully trying to achieve a 
cognitive goal (Scardamalia & Bereiter, 1993/94; Willis & Tucker, 2002). 


In 1 999, the National Council for the Accreditation ofTeacher Educa- 
tion (NCATE) issued guidelines that associated portfolios with electronic 
versions of portfolios (Aschermann, 1999), resulting in NCATE- accred- 
ited teacher education programs in the United States taking a more focused 
look at how to deal with the electronic aspect of portfolio development, 
storage, and presentation. Research on electronic portfolios has focused 
on implementation issues, and empirical research addressing the effects of 
electronic portfolios is sparse. Zeichner and Wray (2001) voice the same 
concern: “Despite the current popularity of teaching portfolios, there have 
been very few systematic studies of the nature and consequences of their 
use for either assessment or development purposes” (p. 615). 

An electronic portfolio is collected, saved, and stored using the Web 
(Barrett, 1998). Electronically, a portfolio can be shared among multiple 
parties simultaneously and can contain multiple media. Systems have been 
implemented that support the electronic portfolio in teacher education 
programs. The comparison of paper portfolio and electronic portfolio in 
surveys of students showed that students prefer the electronic portfolio 
(Carney, 2001; Franco, Hendrick, Huston, & Kim, 2004). However, 
no model or system has existed that combines the electronic portfolio 
learning, the alternative/ traditional assessment vehicles, and the Student 
Information System. Combining and integrating these components might 
help students, teachers, and administrators manage the educational pro- 
cess, improve organizational operational efficiency, and improve learning 
outcomes for students. No such combined system has been described in 
the literature, and this study will present a model for such a system. 

This paper presents the findings from two data sources: attitudinal 
data and cognitive data. Carney (2004) suggests that such attitudinal 
(self-reported) and cognitive (learning outcome) data are vital to under- 
standing electronic portfolios. First, the Web-based Accountability Model, 
the software, and the electronic portfolio process are introduced. Second, 
teacher candidates’ views towards using the software are examined. Third, 
the results of a content analysis study reveal how the teacher candidates’ 
cognitive skills developed through using the software. Finally, conclu- 
sions are drawn on how the model addresses accountability and could 
be generalized in other teacher education programs. 

Web-based Accountability Model 

The Department of Education (hereafter called the Unit) at the Uni- 
versity of Maryland, Baltimore County offers certification programs at the 
graduate and undergraduate levels in Early Childhood Education (P-3), 
Elementary Education (1-8), and Secondary Education (7-12), and at the 
graduate level but not the undergraduate level in English for Speakers of 
Other Languages (ESOL). All undergraduate teacher candidates complete 
a major in an academic area and a certification program. 

In the year 2000, the Unit started developing plans for an assessment 
system to meet the NCATE 2000 standards (NCATE, 2000). Historically, 
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Table 1: Levels and Dimensions of Web-based Accountability Model 

Effect 


^^\Dimension 

Level 

Learning 

Assessment 

Individual 

Candidates' learning 
process and outcomes 
affected by the Web-based 
Accountability Model 

Triadic assessments on 
candidates' performance 

Organization 

Organizational learning 
regarding operation 
efficiency and 
effectiveness 

Triadic assessment on the 
Unit's operation efficiency 
and effectiveness 


the Unit relied on a combination of formal (course grades, test grades, 
planned observations) and informal assessment (anecdotes and gradu- 
ate reports). However, the process was time consuming and sometimes 
inaccurate. Many resources were consumed in gathering paper data to 
analyze and draw conclusions from that data. The Unit’s four academic 
programs each had peculiar assessment features. Unit- wide assessment was 
awkward. The demands for accountability from the state and the federal 
governments and NCATE dictated more systematic assessment. 

Web-based Accountability Model 

A Web-based Accountability Model was conceptualized in 200 1 . The 
implemented information system, which was called the Education Ac- 
countability Systems (EAS), was put into use in 2002. The Web-based 
Accountability Model balanced two dimensions: the learning and as- 
sessment dimensions. The design also represented accountability at two 
levels: Unit (or organization) level and Candidate (or individual) level. 
The Unit was held accountable for preparing the content knowledge 
and instructional strategies and supporting candidates to meet teacher 
licensure standards and develop professionally. Candidates were account- 
able for meeting standards set by the Unit and demonstrating learning 
outcomes. The learning and assessment dimensions interacted with the 
unit and candidate levels. (See Table 1.) 

The Web-based Accountability Model had three components: 

• Electronic portfolio to facilitate candidates’ learning and help advi- 
sors assess candidates’ learning, 

• Performance assessment to assess candidates’ learning and program’s 
operation, and 

• Accountability Center to contain student information (such as 
transcripts) and aggregated data from performance assessment, to 
generate reports based on the aggregated data for program adminis- 
trators. 

The information flow among them integrated assessment and learning 
into one organic cycle. (See Figure 1 .) 

Multiple-sourced performance assessment provided the Accountability 
Center with assessment data to generate guidelines for interventions and 
improvements in the candidates’ learning and teaching process. The Web- 
based Accountability Model was based on standards from: 

• National organizations (such as NCATE, INTASC Principles), 

• State guidelines (such as Maryland Redesign of Teacher Education), 
and 

• Specialized professional associations (such as NAEYC, ACEI, and 
TESOL). 

These standards had been adapted and embedded in the electronic 
portfolio and performance assessment. Performance assessment included 
the electronic portfolio assessment, field experience assessment, clinical 


Accountability 



Figure 1: Web-based Accountability Model Conceptual Model: Arrows 
indicate information flow. 

practice performance assessment, Unit and program assessments, and 
post-graduation assessment. 

The three production components of the EAS enabled the Unit to 
generate and evaluate both qualitative and quantitative performance data 
at each of five benchmarks: 

• Program entrance: admission criterion such as passing PRAXIS I, 
online application, etc. 

• Course and Field Experience: foundation and method courses 
evaluation, early field experience assessments, etc. 

• Clinical Experience: seminars and 100-day clinical practice, course- 
work evaluation, etc. 

• Program Exit: pass PRAXIS II, post-clinical practice survey, licen- 
sure application, etc. 

• Post Graduation: professional portfolio, alumni/employer survey, 
etc. 

At the Unit level, Unit-centralized performance assessment instru- 
ments were used to assess how well the Unit was working toward its goal 
of preparing caring, knowledgeable, thoughtful, and skilled teachers. 
These assessment results were transformed into organizational learning for 
further improvement. At the candidate level, the electronic portfolio show- 
cased how candidates integrated what they learned into their internship 
teaching to meet academic requirements. More detailed information about 
the model and the system was delineated by Ma and Rada (2005). 

Electronic Portfolio Process 

The internship consisted of two phases — Phase I and Phase II. This 
year-long experience took place in the University’s Partner Professional 
Development Schools (PDS). The Phase I internship was a one- or two- 
day-per-week placement in a PDS that was linked to specific methods 
courses within programs. Phase II was the full-time teaching experience 
where teacher candidates were engaged in the activities of planning, 
implementing, and evaluating learning. As a pilot study, only Phase 
II interns were required to complete an electronic portfolio because 
this phase provided teacher candidates extensive internship experience. 
The electronic portfolio and performance assessment sub-systems were 
introduced to the candidates at the end of the semester before starting 
their Phase II internship. They were required to construct an electronic 
portfolio to document knowledge, skills, dispositions, and learning process 
to demonstrate standard-based proficiencies. (See Figure 2.) Additionally, 
interns, mentors, and supervisors conducted performance assessments to 
reflect interns’ learning progress and outcomes. 


112 Journal of Computing in Teacher Education Volume 22 / Number 3 Spring 2006 






Introduction 


ConMcl 


Logout I 


Figure 4: Electronic portfolio: Reflection Editor 

All interns enrolled in an internship seminar at the beginning of their 
internships. They were taught how to develop an electronic portfolio with 
hands-on training workshops provided by the Unit’s professional support 
team. At the end of their internships, each candidate created a Showcase 
electronic portfolio for a formal presentation in front of a review team, 
which consisted of at least one supervisor, one mentor teacher, and one 
faculty member. The electronic portfolio was reviewed using a set of 
predefined rubrics for each teacher standard. 

In the course of constructing their electronic portfolio, candidates 
created artifacts in the form of documents, photos, graphics, audios, and 
videos. The electronic portfolio provided a standard-based framework for 
assessing candidates’ learning. It had three essential functions: evidence 
collection, selection, and reflection. (See Figure 3.) Evidence was collected 
from the actual P-12 teaching activities. Other than providing broad 
guidelines about what artifacts to include in the electronic portfolio, the 
Unit left specific decisions about the choice of artifacts exclusively to 
the teacher candidates. The collection and selection of the evidence ad- 
dressed the question “What did Ido?” (Van Wagenen & Hibbard, 1998). 


The reflection answered the questions: “What did I learn?,” “How does 
this evidence support the learning?,” and “What will I do next?” (Van 
Wagenen & Hibbard, 1998). The electronic portfolio provided a Web 
editor accompanying each standard so that candidates could enter their 
reflections adjacent to the standard. (See Figure 4.) 

Survey Study 

Two levels of effect of introducing the EAS into the Unit were analyzed: 
candidate effect and organization effect. Only candidate effect was ex- 
amined in this paper. First, teacher candidates’ views and beliefs towards 
using the EAS system were tested. Second, the effect of EAS usage on 
reflective and cognitive activities was explored. 

Subjects 

Sixty-two teacher candidates were doing internships in the teacher educa- 
tion program in the spring of 2002. Each candidate was paired with one 
or two mentor teachers in the local schools where they performed the 
internship, and one supervisor who was either from the Unit’s faculty or 
from local schools. Each intern was in one of four programs: Early Child- 
hood, Elementary, ESOL, and Secondary. The subjects for the survey study 
were drawn from these four strata. The four strata, with the total number 
of potential subjects in parentheses, are: Early Childhood (6), ESOL (8), 
Elementary (20), and Secondary (28). Proportional stratified sampling 
was used to generate a sample size of thirty-one subjects. (See Table 2.) 
The subjects were selected through a systematic random sampling of fifty 
percent of the total population for each individual stratum. A total of 
thirty-one candidates were thus chosen for this study. 
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Figure 3: Electronic portfolio: Selected Artifacts and Accompanying 
Reflective Statements 


Table 2: Name and Total Number of Each of the Four Strata along with 
Number of Sample 
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Figure 2: Electronic portfolio: Standard-Based Reflection Framework. 
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A^|act One Seating Chari 

This sealing chad was a crucial management tool for my middle school 
classroom My mentor teacher and I designed a team management system lor 
our classroom The children sit m teams Each team can earn frve points a 
day The breakdown of the points is one point for attendance of everyone at the 
table, one point for agendas, and one point for homework There are also two 
behavior points If the behavior pomts are lost it is possible to earn them back 
by the end of class. The children work as a team lo earn these points This 
keeps them accontable with their peers, and working togeiher The points are 
added up at the end of class and at the end of the week the team with the most 
points is rewarded 
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Artifact Two: Bulletin Board 

I created this bulletin board with the help of another student intern at the 
beginning of the school year at Van Bokkelen This board showcases a reading 
incentive program for the students Each students and teacher has a basket 
Whenever a student reads ten books, he earns one apple Ten apples in the 
basket earns the student an invitation to an ice cream or pizza party at the end 
of the school year This board is intended to be an individual and group 
motrvator Indmduals are responsible for their own baskets and students are 
motivated by the progress of their fhends and teachers as well. 
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In my interpretation, the first INTASC standard addresses the need for the teacher to 
understand what he or she teaches This standard says that the teacher needs to know about 
the discipline that they teach, and they need to know how to teach it This is a fundamental 
principle to becoming a good teacher As a future engfash teacher, if I were not familiar with the 
subject of language arts. I would not become a good language arts teacher This standard also 
points out that the teacher has to always be on the quest for more knowledge in the subject 
area(s) that she teach A teacher is always a learner Finally, the teacher should have such a 
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Strata 

Population (N) 

Sample (n) 

Early Childhood 

6 

3 

ESOL 

8 

4 

Elementary 

20 

10 

Secondary 

28 

14 



















Table 3: Pre- and Posttest of EAS Belief Survey Results from 31 Subjects during their Internship 


Survey Questions / Hypotheses 

Pre M 

Pre SD 

Post M 

Post SD 

Paired 

t-test 


Time constraints were my main concern in developing electronic portfolio 

3.60 

.503 

3.05 

.686 

.002* 


Electronic portfolio increased my technology skills 

4.05 

.510 

4.25 

.550 

.163 


Electronic portfolio increased my awareness of effective technology integration into classroom 

3.90 

.447 

4.15 

.489 

.056 

O 

Electronic portfolio increased my content knowledge 

3.80 

.523 

4.05 

.510 

.056 

o 

Q. 

O 

Electronic portfolio helped me meet MTTS Standards 

3.85 

.489 

4.55 

.605 

.003* 

'c 

o 

o 

Electronic portfolio helped me meet INTASC Principles 

4.00 

.562 

4.50 

.513 

.002* 

_QJ 

LU 

Electronic portfolio will be useful to me after graduation 

3.90 

.308 

4.05 

.394 

.083 

CD 

Performance assessment helped my self-assessment 

3.75 

.444 

3.95 

.510 

.104 

1 £ 

Performance assessment facilitated my learning to teach 

4.05 

.510 

4.70 

.470 

.000* 

~ C/5 

£ 8 
5s 8 

Q_ CD 

Unit made fair assessment on my learning and teaching based on information collected by the 
performance assessment sub-system 

4.20 

.523 

4.35 

.489 

.267 


*P< .05 

l\lote: 1=Strongly Disagree, 2=Disagree, 3=Not Sure, 4=Agree, and 5=Strongly 
Agree. Pre M means Mean of the Pretest and Pre SD means Standard Deviation 
of the Pretest. 

All teacher candidates participating in this study were required to have 
an electronic portfolio. Only Early Childhood candidates used a paper 
portfolio before using the electronic portfolio. The focus of this explor- 
atory study was the teacher candidates’ views towards using the EAS. 

Instrument and Procedure 

The EAS Belief Survey was developed to measure students’ views toward 
using the EAS. The survey instrument was analyzed by two faculty mem- 
bers in the Unit, as well as two outside experts in the same field, to de- 
termine whether the questions were valid. The final instrument consisted 
of 10 questions divided into two separate sections. Section I addressed 
candidates’ views towards using the electronic portfolio sub-system and 
section II addressed candidates’ views towards using the performance 
assessment sub-system. Section I had seven questions, including: “Elec- 
tronic portfolio increased my technology skills,” “Electronic portfolio 
increased my content knowledge,” and “Electronic portfolio helped me 
meet INTASC Principles.” Question two had thirteen sub-questions, 
which were adapted from the survey by Barrett (2003) to evaluate mul- 
timedia skills gained by developing an electronic portfolio. (See Table 4.) 
A respondent would check one item if he/she considered the electronic 
portfolio development has helped him/her to gain such a skill. Section II 
had three questions: “Performance assessment helped my self-assessment,” 
“Performance assessment facilitated my learning to teach,”, and “Dept, 
made fair assessment on my learning and teaching based on information 
collected by the performance assessment sub-system.” The EAS Belief 
Survey used a Likert scale ranging from one (Strongly Disagree) to five 
(Strongly Agree). 

In week one, candidates were introduced to electronic portfolio 
development and completed an assigned task in electronic portfolios. In 
week two, the EAS Belief Survey was administered among twenty subjects 
as a pretest. Then after ten weeks of electronic portfolio development, 
the EAS Belief Survey was again given to the subjects for the posttest. 
Thirty-one preservice teachers’ responses to the EAS Belief Survey were 
analyzed using the paired sample r-test. 

Results 

All subjects submitted the pre- and posttest of the EAS Belief Survey; 
responses were reported with mean scores and standard deviations. (See 


Table 3.) Comparisons between pre and post results of each question 
were analyzed using dependent samples r-tests. Each question was treated 
separately as an individual null hypothesis in the analysis. Based on the 
p value of r-test, results were discussed in two groups: significant and 
nonsignificant. Then the discussion about two groups was provided. 

Significant Results 

The response to the question “Developing electronic portfolio helped me 
to meet Maryland Teacher Technology Standards” was significantly dif- 
ferent between the pre- and posttest (t = 3.390, p < .05). The significant 
increase suggested that the preservice teachers gained knowledge and 
understanding of the teacher standards that they had not fully anticipated 
prior to specifically using electronic portfolio to address them. The same 
rationale applied to the item “Developing electronic portfolio helped me 
to meet INTASC Principles” (t = 3.684, p < .05). It also indicated that, as 
a direct result of this experience, the preservice teachers felt that meeting 
teacher standards mandated by nation and state were greatly facilitated 
by the development of the electronic portfolio. More important, their 
responses also indicated an increased acceptance of using these standards 
to guide their teaching practice. This supported the theory of embed- 
ding teaching standards into the electronic portfolio, which was one of 
the important principles underpinning the development of electronic 
portfolios in the Web-based Accountability Model. 

Another significant difference was found between pre and post re- 
sponses to the question “Performance assessment facilitated my learning 
to teach” (t = 4.333, p < .05). Pre survey responses to this statement were 
comparatively low (M = 4.05, SD = .510), but post survey responses 
showed an increase in perceived importance (M = 4.70, SD = .470). 
Performance assessment was designed to deliver timely feedback to can- 
didates. Candidates considered feedback from supervisors and mentors 
very helpful. The timely feedback was provided by the dynamic assessment 
report module featured in the performance assessment sub-system. The 
feedback reminded candidates what competencies were required of the 
profession of teaching. The performance assessment might help teacher 
candidates adjust their learning and teaching process, by which they 
achieved the competencies to be a teacher. 

The final item showing a significant difference between pre and post 
results was the item regarding the “time constraints of the electronic 
portfolio development”, which decreased measurably (t = -3.584, p < 
.05) from pre (M = 3.60, SD =.503) to post (M = 3.05, SD = .686) 
survey. Prior to constructing electronic portfolios, the preservice teach- 
ers reported concerns over time constraints in completing electronic 
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Table 4: Multipart Technology Skills Sub-Questions 


Multipart Technology Skills Sub-Questions: This table shows the technology skills in the leftmost column, the range of responses in the upper row, and the percent of 
candidates choosing a response in the cells of the table. 

Multipart Questions of Technology Skills 

Before 

After 

Do not know 

No 

Response 

Set up folders to organize files on computer hard drive 

93 

0 

0 

7 

Use the File Transfer Application to transfer files between University, school, and home. 

3 

3 

94 

0 

Use Web Page Editor such as Microsoft FrontPage /Macromedia Dreamweaver 

19 

10 

61 

10 

Use advanced features of Microsoft Word (Document Map, Hyperlinks, etc.) 

42 

29 

19 

10 

Use advanced features of Microsoft PowerPoint (slides animation, drawing, etc) 

42 

29 

26 

3 

Use a digital camera to take pictures 

74 

10 

16 

0 

Scan images with a desktop scanner 

29 

29 

19 

23 

Edit images such as resizing/cropping 

10 

10 

51 

29 

Transfer video to a computer for editing (in lab) 

0 

3 

78 

19 

Edit a digital video on PC/Mac 

0 

0 

90 

10 

Record digital audio to the computer with a microphone 

52 

3 

0 

45 

Create and edit Adobe Acrobat files from a variety of computer applications 

23 

16 

39 

22 

"Burn" a CD/DVD 

39 

23 

23 

23 


portfolios. However, by the completion of electronic portfolios, it was 
much less of a concern. Electronic portfolio completion, in itself, may 
have alleviated their concerns about having enough time for completion. 
However, this finding may also suggest considerations beyond the obvi- 
ous. For example, the preservice teachers’ concerns about their ability to 
complete electronic portfolio in the provided time may be an indirect 
result of their apprehensions about working with this tool for the first 
time, tight schedule of student teaching, uncertainty in their technology 
skills, or a variety of other factors that served to reduce their confidence 
and inversely increase their concerns over completing the electronic 
portfolio in the allocated time. 

Nonsignificant Results 

No significandy different outcomes were found to the question “Electronic 
portfolio increased my technology skills.” The multipart technology skills 
questions in the survey, which asked about the change in technology 
proficiency, showed that preservice teachers had already mastered some 
of the technology used in this pilot study, such as word processing and 
digital photographing. (See Table 4.) Other factors accounting for this 
result included: 

• First, the electronic portfolio provided built-in templates. On one 
hand the templates made the electronic portfolio easy to use; on 
the other hand, the templates possibly discouraged candidates 
from developing the technology skills that were embedded in the 
templates. A case in point could be the low percentage of skill-gain- 
ing using a web page editor. (See Table 4.) The electronic portfolio 
rendered content using a pre-selected template. Therefore, the Web 
editing tasks were almost invisible to the candidates who only filled 
in blanks without setting font or color. 

• Second, the portfolio lab staff reported spending most of their time 
doing things for the candidates. Although it was their job to help 
candidates overcome technology obstacles, they did much of the 
technology tasks for the candidates. When asked why, one staff 
member said it would be faster to do it herself than teaching the 
candidates how to do it. 

• Third, some technologies listed in the survey were not required to 
use in this pilot study. 

Therefore, further study was needed to examine the effect of electronic 
portfolio construction on the development of technology skills. 


Responses to the question “Electronic portfolio increased my content 
knowledge” did not show significant difference between pre- and posttests. 
Although the evidence collected from the electronic portfolio showed that 
learning took place through candidates’ teaching practices, most of the 
candidates felt that they already learned enough about the subject matter 
from the previous courses. Accordingly, the electronic portfolio develop- 
ment was not considered a way to acquire content knowledge. 

The question “Electronic portfolio will be useful to me after gradu- 
ation” reflected candidates’ perspectives of using electronic portfolio in 
the future. The f-test result was not significant. This nonsignificant result 
might be explained in several ways. Some candidates voiced concerns that 
they might not get a teaching position and could switch to another career 
track. Some candidates might have thought that the electronic portfolio 
system would not be readily accessible to them in the future or that their 
colleagues in their future workplaces might not appreciate the usefulness 
of the electronic portfolio. 

No significant difference was found for the question “Performance assess- 
ment helped my self-assessment.” The performance assessment sub-system was 
designed primarily to help unit administrators. The design perhaps needed to 
be reconsidered as regards to helping candidate’s learning process. 

Candidates’ views on the question “Dept, made fair assessment on my 
learning and teaching based on information collected by the performance 
assessment sub-system” were not significantly different in the pre- and 
posttest. The candidates did not consider the introduction of the perfor- 
mance assessment sub-system to significantly affect the Unit’s decision 
making about their teaching and learning. Further study would focus on 
investigating the reliability and validity of the performance assessments 
employed in the performance assessment sub-system. 

Discussion 

The EAS Belief Survey revealed that the EAS could positively facilitate 
candidates to meet teacher standards and learn to teach. The electronic 
portfolio sub-system provided consistent opportunities for the candidates 
to reflect on their teaching experiences and examine the gap between their 
beliefs and real teaching. These were fundamental to the success of their 
concurrent student teaching experiences. The performance assessment 
sub-system was considered to facilitate candidates learning to teach in 
that it provided timely feedbacks to candidates. These feedbacks could 
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Table 5: Analysis Framework: Cognitive Skills (Adapted from Bloom, 1956) 


Cognitive Skills 

Definitions 

Indicators 

Description 

(Bloom's knowledge level) 

Observing or studying a problem: 

Understanding of content, pedagogy of teaching: 
Mastery of subject matter; 

Simply describing the subject matter; 
Identifying relevant element; 
Reformulating the problem; 

Explanation 

(Bloom's comprehension 
level) 

Understanding information: 

Grasp meaning; 

Interpret facts, compare, contrast; 
Order, group, infer causes; 

Predict consequences; 

Understand, explain and interpret education theory; 
EX. Paraphrase a theory rather than simply recite it 

Application 

(Bloom's application level) 

Proposing coordinated actions for the application of a solution. 

Use methods, concepts, theories in new situations; Solve problems 
using required skills or knowledge; 

Inferencing 
(Bloom's analysis and 
synthesis level) 

Induction and deduction; 

Organization of parts; 

Recognition of hidden meanings; 

Identification of components; 

Relate knowledge from several areas; Translate knowledge into 
new context; 

Break down a situation into its component parts; 
Distinguish between facts and inferences; 
Drawing conclusions; 

Making generalizations; 


be accordingly used by the candidates to adjust their teaching strategies. 
The performance assessment sub-system fostered an electronic education 
community including the university and local schools. 

Preservice teachers’ views toward computers and their technology skills 
can be improved by integrating technology into the teacher education 
coursework (Kumar & Kumar, 2003). But the improvement was limited 
if teacher candidates were not provided opportunities to learn and apply 
skills and strategies in real-world settings (Strudler et al., 1998). This study 
has revealed other confounding factors with regards to increasing technol- 
ogy awareness and application. Using the EAS to develop an electronic 
portfolio provided the candidates an opportunity to apply technology 
into real tasks. However, the results of the survey showed that candidates 
did not consider that EAS usage helped them increase technology skills. 
The electronic portfolio development only took one semester for this 
pilot study. The candidates might not have had enough time to develop 
new technology skills in this short time frame. The electronic portfolio 
development might be introduced earlier and across multiple courses. 

A modified approach for using the electronic portfolio was developed 
based on this study’s results. The approach gave candidates more time to 
get familiar with the system and to start thinking about collecting evidence 
from their early field experience. This approach was used in multiple 
methods courses and started in Phase I of the internship. 

Another challenge emerged from this survey was the duplicity of 
purpose embedded in the electronic portfolio development. The tensions 
between balancing the learning focus and the ever-present accountability 
issues continued to exist. Assessment and learning represent two ends of a 
continuum. The EAS, based on the Web-based Accountability Model, was 
designed to integrate both ends trying to strike a balance. As the survey 
results suggested, teacher candidates’ expressed positive attitudes towards 
using the EAS to facilitate learning, while not so positive with regards to 
the assessment. The good thing about it was the performance assessments 
used in the EAS seemed to improve candidates’ learning. This echoed 
what Wiggins (1993) said, “...assessment should be designed to improve 
performance, not just monitor'll (p. 6, emphasis in original). 

Although the views of teacher candidates regarding their use of the 
electronic portfolios system provided an important perspective, as Carney 
(2004) states, we also needed to consider the learning outcomes of can- 
didates who completed electronic portfolios. The next section addressed 
student learning outcomes through a direct measure of the teacher educa- 
tion candidates’ electronic portfolio content. 


Content Analysis 

The specific cognitive skills that candidates demonstrated were categorized 
and measured in a content analysis of their electronic portfolios. The con- 
tent analysis enabled the researcher to search for structures and patterned 
regularities in the text and make inferences on the basis of these regulari- 
ties. The population from which the sample was chosen was the same as 
used for the above survey study. Four candidates were randomly chosen 
from each of four programs for the cognitive skills investigation. 

Instrument and Procedure 

Cognitive skills demonstrated in the electronic portfolios were analyzed 
within a taxonomy framework. Bloom’s (1956) taxonomy for the cognitive 
domain described progressively higher levels of cognitive activity from 
factual information at the knowledge level to judgment and rating of 
information at the evaluation level (Bonk & Sugar, 1998). For this study, 
an analysis framework was adapted and modified from this taxonomy to 
categorize and code the electronic portfolio content into four levels of 
cognitive skills for reflection. (See Table 5.) 

The electronic portfolio content was collected from candidates’ re- 
flective statements for INTASC Principles. Content analysis was used 
to analyze the text. The researcher plus two graduate assistants read each 
posting. The raters read the text and categorized them using the above 
established analysis framework. Two graduate assistants participated in five 
hours of training to learn to code the text. The raters coded independendy 
of each other. The raters coded two sample candidates’ electronic portfolio 
to establish inter-rater reliability. After this initial categorization of the 
data, a 79% agreement rate was found among the three raters. Once 
inter-rater reliability was achieved, four subject candidate’s electronic 
portfolio text was coded by the three raters. Average rater reliability was 
attained at 81%. Conflicts were resolved by having three raters re-exam- 
ine the paragraph in question until 100% agreement was reached. Some 
paragraphs, however, clearly demonstrated more than one cognitive skill 
and were coded as such. 

Results 

Content analysis of subjects’ electronic portfolio reflective statements 
demonstrated subjects’ cognitive skills development with the help of 
electronic portfolios. As stated above, the analysis framework identi- 
fied four categories related to cognitive dimension: (1) description, (2) 
explanation, (3) inferencing, and (4) the application of strategies. All of 
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Table 6: Percentage of Each Cognitive Skill Type Exhibited in Four 
Subjects' Electronic Portfolios Over Time 


Checkpoint 

Description 

Explanation 

Inferencing 

Application 

Week 3 

39% (9) 

30% (7) 

17% (4) 

13% (3) 

Week 6 

29% (14) 

32% (15) 

21% (10) 

18% (9) 

Week 10 

25% (21) 

36% (30) 

20% (17) 

19% (16) 

Avg. 

31% 


20% 

17% 


Note: The number of occurrences is indicated in parentheses; the bottom 
row indicates the average percentage over three checkpoints. The computed 
numbers are to two significant digits and the percents in a row do not 
necessarily sum to 100%. 

these cognitive skills were identified in candidates’ electronic portfolio 
reflective statements. The total occurrences of each cognitive skill in all 
subjects’ writings were summed and converted to percentage to present 
an overall pattern. (See Table 6.) 

Description of a technical or theoretical aspect of teaching appeared an 
average of 31% of the time in subjects’ completed electronic portfolios. 
An example of this type of reflection would be: 

A chief principle of ESOL education is to teach material 
that is comprehensible and interesting to students ... 

To help children learn, teachers need to develop a solid 
understanding of the way in which they grow, both 
intellectually and personally. 

This type of reflection stated that preservice teachers’ opinion of a 
specific teaching theory or activity, but did not justify or interpret it for 
any specific behavior. 

The most frequent type of cognitive reflection was explanatory content 
(average at 33% over the time). Its percentage among all cognitive skills 
demonstrated in the reflection has been steady over time. An example of 
this type of reflection would be: 

...Keeping an open, empathetic mind can help a teacher 
to bond with a student or group of students, and will 
enrich instruction for all. 

This type of reflection suggested that candidates saw the logic behind 
theory and could rationalize the success or failure of the instruction. Much 
of this type of reflection helped candidates comprehend how teaching 
theory could be applied during practice. 

Reflections that addressed the inferencing aspect of teaching accounted 
for 20% of occurrences over three checkpoints in all identified text. An 
example of this type of reflection would be: 

Observation and assessment is essential to understanding 
young children’s development. Having a strong partner- 
ship with families and other professionals can positively 
influence the child’s learning. Parent input can be very 
helpful with informal assessments. The child’s parent may 
be seeing different things then you are seeing. 

This kind of reflection showed that candidates made connections 
between concepts from coursework and internship teaching experiences. 
It encouraged preservice teachers to create their own teaching philosophy 
and incorporate effective teaching strategies into teaching practice. 

Additionally, the application level of reflection occurred in 17% of 
coded terms over three checkpoints. (See Figure 5.) An example of this 
reflection would be: 



In October, I was observed and my supervisor noted 
that due to the students’ inability to exercise self-control, 

I needed to exhibit stronger classroom management 
procedures. I reflected on this experience and greatly 
changed my demeanor in the classroom to enforce con- 
sequences while also frequently praising students for 
positive behavior. 

The electronic portfolio provided the opportunity for preservice teach- 
ers to document when and how theory was realized through practice. 
This could be demonstrated by its consistent increase over time. (See 
Figure 5.) 

Discussion 

In week three, candidates had developed their electronic portfolios for 
two weeks to collect artifacts and reflect on their internship. The cogni- 
tive skill of description appeared most frequently in this week (See Table 
6), followed by explanation, inferencing, and application. As develop- 
ment of electronic portfolios came to week six, the changed distribution 
of cognitive skills revealed a transformation from being descriptive to 
being explanatory, which continued to week ten. Also, candidates were 
engaging in more application level cognitive activities at this stage. The 
electronic portfolio was successfully accepted by candidates as a cognitive 
tool to document and reflect candidates’ learning and teaching process. 
One candidate commented, “Writing explanations and rationales gave 
me a chance to see what I really knew and to see how far I’ve come since 
starting the education program. It was a tool for me — to reflect.” Further 
work should address in more detail what are the focus of these reflective 
statements, be it technical, situational, or sensitizing (Tsangaridou & 
O’Sullivan, 1994). 

The total percentage of cognitive skills at application level increased 
steadily from 13% at week three to 19% at week ten. (See Figure 5.) 
Overall, inferencing type of reflection (i.e., drawing conclusions and 
forming generalizations) stayed almost unchanged. The longitudinal 
content analysis confirmed that keeping their reflections organized in the 
electronic portfolio enabled candidates to reflect and revise continuously. 
The electronic portfolio gave candidates the advantage of easily seeing 
how their standard-meeting activities changed over time, which would 
be hard for a paper portfolio to capture (Aschermann, 1999). 

The content analysis also confirmed the findings discussed in the 
survey study. Candidates reported learning the pedagogy by using the 
EAS. The electronic portfolio reflection served as a story-telling tool 
and enabler for candidates to learn from the teaching experiences. This 
was consistent with the literature suggesting portfolio development may 
support reflection. 
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Conclusion 

This study proposed employing an electronic portfolio and performance 
assessment for accountability purposes in a teacher education program. A 
Web-based Accountability Model was established and an EAS was success- 
fully implemented based on this model. The Web-based Accountability 
Model represented a cycle of learning and assessment at both individual 
(teacher candidates) and the organizational (Unit) level. The Web-based 
Accountability Model combined assessment results from question-based 
instruments and electronic portfolios to direct the interventions to teacher 
candidates’ learning process. 

The findings of the survey and content analysis study were consistent 
with the literature that portfolio development could facilitate learning. The 
self-reported survey data were triangulated with the content analysis results 
to demonstrate that the electronic portfolio could help candidates learn 
to teach. The electronic portfolio provided an opportunity for candidates 
to collect artifacts in multiple formats, modify artifacts with the develop- 
ment process, critically examine their practices by reflection, and connect 
evidence to standards. In particular, the electronic portfolio facilitated 
candidates’ understanding of standards because EAS emphasized standards 
criteria. Content analysis confirmed that the candidates’ cognitive skills 
were improved in the electronic portfolio development course. 

Based on the findings in this pilot study, the EAS was considered use- 
ful for Standard 2 of the NCATE accreditation requirements: assessment 
system and unit evaluation. The electronic portfolio together with per- 
formance assessment helped the Unit make changes based on candidates’ 
data. The EAS played an important role in reshaping the Unit’s processes 
and operations. Summaries of data on candidate performance, program 
quality, and Unit operations were reported to and interpreted by the Unit’s 
faculty. The Unit’s NCATE Steering Committee conducted Unit-level 
reviews and made recommendations for change. The Unit’s Assessment 
Committee implemented the change and monitored impact. As a result, 
the Unit changed candidates’ internship requirements, created multiple 
checkpoints based on the electronic portfolio and performance assessment, 
and re-organized the Unit’s supporting and administering facilities. 

Future research will address various concerns. The survey results 
suggested that further research was needed. For instance, candidates did 
not report gaining technology skills in this study. This might be due to a 
variety of factors, such as laboratory assistants performing complex tasks 
for candidates and candidates not having enough time to explore new 
technology. Some other worthy research directions are: What is the orga- 
nizational effect of the EAS? What degree of balance between assessment 
and learning is appropriate to maximize teacher candidates’ self- regulated 
learning and reflective teaching while serving accountability purposes? 
Conducting longitudinal studies on relationships among assessments, 
licensure examinations, and on-the-job performance will provide further 
evidence and guidance for program quality improvements. 
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